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better organized. Students were likely to talk with parents or peers 
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Assessing the Impact of the Early Alert Program 



Abstract 

Undergraduate students need feedback concerning their performance in class if they are to 
improve. The Early Alert program was created as a pilot project whereby undergraduate students in 
specific courses were sent an "Early Alert" notice if they were receiving a C- or below at the five week 
point of the semester. The focus of this study was to identify how students reacted to the Early Alert 
notice and to substantiate whether students who received the EA notice actually performed better than a 
group of similar undergraduate students who served as a control group. Findings and implications are 



discussed. 



Introduction 



Undergraduate students need feedback concerning their performance in class if they are to 
improve (Chickering & Gamson, 1987, 1991). This becomes especially true for new students who may be 
doing poorly, as it suggests that adjustments or assistance may be necessary to enhance their standing by 
the end of the semester. At the present institution, undergraduate students do not receive mid-term grades. 
Further, there is no formal mechanism in place that would inform the student that they are doing poorly in 
a course. Consequently, the Early Alert (EA) program was created as a pilot project whereby 
undergraduate students in specific courses were sent an "Early Alert" notice if they were receiving a C- or 
below at the five week point of the semester. The focus of this study was to identify how students reacted 
to the EA notice and to substantiate whether students who received the EA notice actually performed 
better than a group of similar undergraduate students who served as a control group. 

Background 

In the last decade there have been significant efforts to enhance undergraduate teaching and 
learning at our colleges and universities. Undoubtedly, Chickering and Gamson's (1987, 1991) Principles 
for Good Practice in Undergraduate Education has provided some continuity for this effort. Accordingly, 
one common and well-supported recommendation suggests that faculty members should provide students 
with feedback on their performance during the semester so that students can take appropriate action, if 
necessary, to improve their standing. From a theoretical and practical perspective, this principle has wide 
support (see Sorcinelli, 1991). Moreover, recent research has indicated that strong normative support for 
providing undergraduate students feedback on their academic progress is supported among faculty 
members regardless of their institutional affiliation (Braxton, Eimers, and Bayer, 1996; Eimers, Braxton, 
and Bayer, 1998). Despite the logical support for this recommendation, however, empirical evidence that 
substantiates its effects on academic progress is not widespread. 
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Research Questions 

Consequently, two research questions frame the focus of this inquiry: 

1 . What were students' reactions to receiving an EA notice? What behavior changes, if any, did they 
make? 

2. Most importantly, how did their performance in the class compare to a similar group of students who 
were also doing poorly in the class but did not receive an EA notice? 

Methodology 

The context for the study was a public, research I university in the Midwest. Courses selected for 
the EA program were general education courses geared mainly toward freshmen students. In addition, the 
courses were purposefully selected because historically these courses included a relatively large 
percentage of students who did not pass the course. Once these courses were identified, the faculty 
members teaching these courses were asked if they would participate in the project; faculty members 
teaching 19 sections volunteered. Faculty members participating in the study also promised to provide 
some form of evaluation (and accompanying grade), such as a performance, project, quiz, examination, or 
paper, by the end of the fifth week. Consequently, during the sixth week of class all students who were 
receiving a C- or below in the participating classes were sent an EA notice from the registrar's office. The 
EA notice not only included the student's grade, but also various suggestions for improving performance 
(e.g., you may want to discuss this with your professor, seek help at the academic performance center, 
talk to your advisor, etc.). The student's advisor also received a copy of the notice (see Appendix for a 
copy of the Early Alert form). There were 816 students who received an EA notice. 

Web-based Survey 

The survey was developed by the University Council for Advisement and refined prior to being 
posted on the web. It was designed to elicit responses to two key concerns: 1) What actions, if any, were 
taken as a result of receiving the Early Alert notice? And 2) How did you feel when you saw the Early 
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Alert notice? For both questions a series of responses were provided for the respondent to check. In 
addition to some demographic questions, an open-ended question was provided: In your opinion, how do 
you think the Early Alert system could be improved? A copy of the Early Alert Survey is included in the 
Appendix. 

Approximately four weeks after students had received their Early Alert notices, an e-mail letter 
was sent to the 816 participants. Included in the letter was the website address that included the Early 
Alert survey. The students were asked to go to the website, enter and verify their Student ID, and 
complete the survey. Five days later, a follow-up e-mail letter with the survey web-site location was again 
sent to all non-respondents. A final e-mail message was sent about ten days after the first notification to 
the remaining non-respondents. In this final case, the survey was embedded in the e-mail message (the 
website address was not included in the e-mail) and students were asked to complete the survey and 
return the message via e-mail. 



Response Rate and Bias 

The response rate for the survey was 29.5% (241/816). Although this response rate is slightly 
lower than what is typically found in similar studies, it is quite positive considering the circumstances. 
First, this survey was administered in the two weeks before Thanksgiving break — an extremely busy time 
for students. Second, this is a sensitive subject, particularly for first-time freshmen who represented the 
majority of the subjects. Third, it is possible that the response rate is higher than 29.5% because it is 
difficult to determine whether everyone received and opened the e-mail message. That is, some students 
may not use e-mail or may use a different e-mail account than the one assigned by the University. 



To check for response bias, respondents were compared to non-respondents on five 
characteristics: gender, student level, ethnicity, course enrolled, and Early ^lert grade. Females and 
African-Americans were significantly (p < .01) more likely to respond to the survey than males and other 
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ethnic groups, respectively. Although there were no statistically significant differences in how males and 
females responded to several items on the survey, there were some differences based on ethnicity. 
African-American were more likely to talk to peers and family about their Early Alert notices (p < .01) 
and were also more likely to be “concerned about losing their scholarship aid” (p < .01). Thus, responses 
to these items may be slightly over-represented. In terms of the differences between those who responded 
to the web survey in contrast to the e-mail survey, there were no statistically significant differences. 

Results 

There were 241 students who responded to the survey. Of these, 219 remembered receiving the 
Early Alert notice and consequently are the focus of the study. The first question asked respondents: 
“What actions did you take as a results of receiving the Early Alert Report?” The results of this question 
are noted in Table 1. 

Students were most likely to take personal actions such as studying more or getting themselves 
better organized. They also were likely to talk with parents or peers regarding their Early Alert notice. 
About one in twelve stated that they did not engage in any of the actions that were identified and one in 
fifteen said they withdrew from the course. 
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Table 1. Percentage of Respondents Who took the Following Action as a Result of the Early Alert 
Notice (N=219) 



Rank 


Item 


Action (percentage) 


1 


A 


Study More (85.6%) 


2 


B 


Make yourself better organized (53.4%) 


3 


C 


Talk to Parent/Guardian (47.0%) 


4 


D 


Talk to Peers (44.7%) 


5 


E 


Attend scheduled review and/or study sessions (37.4%) 


6 


F 


Participate in a study group of peers (28.3%) 


7 


G 


Contact Advisor/Course instructor/coordinator (27.9%) 


8 


H 


Party less on School nights (21.4%) 


9 


I 


Party less overall (13.7%) 


10 


J 


None of the above (8.2%) 


11 


K 


Cut back hours of employment (6.8%) 


11 


L 


Withdraw from the Course (6.8%) 


13 


M 


Contact Learning Center (4.1%) 


14 


N 


Contact Academic Retention Services (3.7%) 


14 


0 


Hire a tutor (3.7%) 


16 


P 


Contact Counseling Services (2.3%) 


17 


Q 


Contact Dean's Office (1.8%) 


18 


R 


Contact Disability Services (1.3%) 


19 


S 


Contact Career Counseling (0.9%) 


19 


T 


Contact Total Person Program (0.9%) 


19 


U 


Enroll in Study Skills Course (0.9%) 


22 


V 


Writing Intensive Tutorial Service (0.0%) 



The twenty-two items in Table 1 were grouped into six composite variables that represented actions 
that could be considered to be similar in nature. For example, the items “study more,” “make yourself 
better organized,” “attend scheduled review and/or study sessions,” “participate in study groups,” and 
“enroll in a study skills course” were combined to form the composite variable, “Take Personal Action.” 
The six composite actions, in order of frequency, are noted in Table 2. As a result of the Early Alert 
notice, most students took some “personal action” (e.g., studied more), followed by “talking to peers or 
parents” and then “consulting a professional” (e.g., contacting the professor, their advisor, the learning 
center, etc.). Interestingly, only about a third of the students contacted their instructor, advisor, teaching 
assistant, or a campus service (e.g., learning center). 



Because female students were more likely to respond to the survey, their responses were compared 
to those of male respondents. Likewise, because African-American students were more likely to respond 
than other ethnic groups, the responses of different ethnic groups were compared. Using chi-square, the 
composite actions from Table 2 were used to test for differences. For gender, there were no statistically 
significant differences between female and male respondents concerning any of the six composite 
variables identified in Table 2. For ethnicity, African-American students were significantly more likely to 
“talk with peers or parents” (p < .01) than respondents of other ethnic groups. This was the only 
composite action where a statistically significant difference was noted with the variable ethnicity. 



Table 2. Percentage of Respondents Who Took the Following Composite Actions as a Result of the 
Early Alert Notice (N=219) 

Rank Items Composite Action (percentage) 

1 A,B,E,F,U Take personal action (87.7%) 

2 C,D Talk with peers or parents (60.3%) 

3 G,M,N,0,P,Q,R,S,Y,V Consult a professional (35.2%) 

4 H,I,K Reduce partying/hours working (31.1 %) 

5 J Nothing (8.2%) 

6 L Withdraw from course (6.8%) 



Table 3 identifies the percentage of respondents who had specific reactions to the Early Alert 
notice. For the majority of respondents, it was a “wake-up call for them to do something.” In addition, 
several respondents thought they might lose their scholarship or be dismissed from class because of their 
poor performance in the course. Just over 6% “didn’t care” and another 5% was “angered that someone 
was looking over their shoulder.” 
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Table 3. Percentage of Respondents Who Had the following Reaction to the Early Alert Notice 
(N=219) 

Rank Reaction (percentage) 

1 It was a "Wake-Up" call for me to do something (62. 1%) 

2 Scared that I might lose my scholarship or other financial assistance (22.3%) 

2 Scared that I might be dismissed from the University (22.3%) 

4 Shocked because I thought I was doing fine (2 1 .0%) 

5 Relieved that someone was paying attention (17.3%) 

6 Worried that my parent/guardian might pull me out of the University (12.8%) 

7 Didn't care (6.4%) 

8 Angered by someone looking over my shoulder (5.0%) 



Did various groups address the items in Table 3 differently? There were no statistically significant 
differences for gender. For student level, however, freshmen respondents were significantly more likely to 
say that the Early Alert notice was a “wake-up call” than those students who where sophomores ( p < 

.01). In addition, African-Americans were significantly more likely to identify that they were “scared that 
they might lose their financial aid” than the respondents of other ethnic groups (p < .01). 

Qualitative Responses 

One open-ended item was included on the survey: “In your opinion, how do your think the Early 
Alert system could be improved?” There were 168 comments regarding this item so just over 75% of the 
respondents addressed the question in some manner. Although respondents felt that there were several 
ways in which the Early Alert program could be improved, in general, the majority were very positive 
about the Early Alert program. Respondents felt that it served as a real “wake-up” call and provided some 
impetus for action. At the same time, a small minority of respondents felt that the University was looking 
over their shoulder and they were alarmed that the University did not think that they could take care of 



themselves. 



Control Group and the Early Alert Group 

In four of the nineteen sections, students who received a C- or below at the five-week point were 
randomly assigned to two groups. The first group (EA Group) received the EA notice. The second group 
served as the Control Group and did not receive EA notices. At the end of the semester, final grades were 
compared between the two groups. The four sections included three sections of general biology and one 
section of microeconomics. 

Chi-square analyses and t tests were used to analyze potential differences between the Control 
Group and EA Group. Chi-square analysis noted no statistically significant differences between the two 
groups in terms of ethnicity (p =.29), gender (p =.88), student level (p =.99), or distribution of grades at 
the five-week point ( p =.37). Furthermore, t tests noted no statistically significant differences between 
the two groups when high school rank ( p =.64) or ACT ( p =.96) were compared. In summary, the two 
groups were literally alike. Because we were most concerned with whether a student could raise his or her 
final grade to a C or better (the University’s definition of "success"), final grades were separated into two 
groups (C or higher = successful; C- or lower = unsuccessful). Of the 118 in the control group, 57 or 48% 
were successful. Of the 139 in the EA group, 44 or 32% were successful (see Table 6). The chi-square 
analysis was statistically significant (A* = 7.4, df= l,p <01). 



Table 4. Mean Scores of Control Group and Early Alert Group 



Control Group (N=118) 


Mean 


S.D. 


Five week grade 






(1 1 -point scale) 


2.0 


1.5 


High School rank 


64.1 


19.3 


ACT 


22.6 


2.9 



Early Alert Group 

Five week grade 
(1 1 -point scale) 
High School rank 
ACT 



(N=139) 




11 



10 



2.0 

65.3 

22.7 



1.5 

20.3 

2.9 



Table 5. Comparison Information: The Control Group and Early Alert Group 



Control Group EA Group 



Female 

Caucasian 

African-American 

Freshmen 

Five-week Grade (greater than D-) 



56.8% 


54.0% 


79.7% 


84.2% 


8.5% 


9.4% 


72.9% 


71.9% 


69.4% 


64.0% 



Table 6. Success Rate of Control Group and Early Alert Group 



Success Rate 
Chi-square: 

{X 1 = 7.4, df- l,p < 01) 



Control Group EA Group 

48.3% (57/1 18) 3 1.6% (44/139) 



Limitations 

The limitations of this study include the following. First, faculty members who participated in the 
study were volunteers. Second, in some cases where the student had not updated his or her address at 
college, the Early Alert notices were sent to the student’s home address. Consequently, we cannot 
determine how many students actually received the Early Alert notice. Third, we anticipated conducting 
several focus groups with some of the students who received an Early Alert notice. Group sessions were 
even scheduled in early December. Due to the poor timing during the semester, however, we only had a 
handful of students who attended the focus group sessions. 

Discussion and Implications 

First, the EA notices were effective in stimulating "action" among students who responded. The 
notices apparently did get students' attention and garnered an action intended to improve performance. 
Based on the relatively few who said they "didn't care," the notice did invoke a reaction that suggested 



students were concerned about their performance. Unfortunately, however, the actions students engaged 
in because of the EA notice were not necessarily the actions that lead to improved academic performance 
(only 1 in 3 improved). Further, and especially because many of these students were freshmen, the EA 
may have scared students to the point that they panicked and were unable to improve their standing. This 
may be a particularly important concern. Students who did poorly in these courses had an average ACT 
score of 22.6 and a high school rank of 64.7%. The average ACT score for incoming freshmen in the fall 
of 1998 was 25.7. Thus, reminding relatively poorer students that they are not only in academic trouble in 
this course but also that they should seek academic assistance may have damaged an academic ego that 
was already somewhat fragile. 

Second, the findings of this study contribute to the literature by suggesting that a single, formal 
notification that informs the student that he or she is doing poorly may not be enough. That is, much of 
the literature recommends that frequent and regular feedback on academic performance should be 
encouraged. Consequently, although the EA program may be a move in the right direction, one may find 
that multiple progress reports throughout the semester are necessary to have a positive effect on 
performance. At the same time, because the Control group was more likely to succeed than the EA group, 
efforts need to taken to ensure that the EA notices are not hindering the performance of students in some 
way. 

Third, we were surprised to find that the EA group did not do as well or even better than the 
Control group. Clearly there are limitations to this study and our findings will need to be substantiated. 
However, implementing the EA program requires significant levels of institutional resources and 
cooperation. To be effective, a process like the EA program whereby a single, formal feedback process 
channeled through the registrar's office will require change. Thus, we are currently exploring ways that 
technology can aid faculty members in their efforts to share academic progress with students. Workshops 
presently teach faculty how to use spreadsheet software to communicate academic progress to several 
students that are enrolled in their courses. That is, we feel the EA program might be a move in the right 



direction but that it needs to be less formal, more frequent, less intimidating, and more closely linked to 
the faculty member who teaches the class. 

Conclusion 

This study reported student reactions to the Early Alert program and examined whether students 
who received an EA notice actually performed better than a similar control group. Although students 
reported that the EA notice helped get them back on track, it apparently had little influence on improving 
their academic performance. The evidence suggests that grade deficiency reminders may be insufficient to 
have a real impact on academic performance and more frequent reminders may be a viable alternative. 
Furthermore, this study raises the issue as to whether deficiency reminders used with freshmen students in 
some way negatively affects their ability to improve their standing in class. 
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University Council on Advisement Survey Results for Early Alert, Fall 1998 

1. Do you remember receiving an Early Alert Grade Report in the middle of this semester 
informing you that you had a C- or below at the mid point of the semester? 

Yes 
• No 

2. If so, what actions did you take as a result of receiving the Early Alert Report? (Check all that 
apply) 

Study More 

Make yourself better organized 
Talk to Parent/Guardian 
Talk to Peers 

Attend scheduled review and/or study sessions 

Participate in a study group of peers 

Contact Advisor and/or Course instructor/coordinator 

Party less on School nights 

Party less overall 

None of the above 

Cut back hours of employment 

Withdraw from the Course 

Contact Learning Center 

Contact Academic Retention Services 

Hire a tutor 

Contact Counseling Services 
Contact Dean's Office 
Contact Disability Services 
Contact Career Counseling 
Contact Total Person Program 
Enroll in Study Skills Course 
Writing Intensive Tutorial Service 

3. How did you feel when you saw the Early Alert grade report? (Check all that apply) 

It was a "Wake-Up" call for me to do something 

Scared that I might lose my scholarship or other financial assistance 

Scared that I might be dismissed from the University 

Shocked because I thought I was doing fine 

Relieved that someone was paying attention 

Worried that my parent/guardian might pull me out of the University 
Didn't care 

Angered by someone looking over my shoulder 

4. Where are you living this semester? 

Residence Halls 

Off Campus 
Fratemity/Sorority 
Parent/Guardian 
Other 

5. In your opinion, how do you think the Early Alert system could be improved? 
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Early Alert Notice (Example) 



TO: Name of Student 

GRADE: C- 

COURSE: General Biology 

The grade shown on the Early Alert Report is based on an assessment of progress in a course at the end of 
the Fifth Week of classes and should serve as a warning that performance needs to improved in order to 
complete the course with a C or above. This report is only an alert. 

This grade report is based on the Early Alert Pilot Program for selected Fall, 1998 courses and is not 
campus-wide. Students with less than 60 hours and enrolled in courses which are included in the pilot 
program will receive the Early Alert Grade Report only if their performance in the course is below a C 
(2.0). Students taking courses which are not in the pilot program who do not receive the Early Alert Grade 
Report should not consider the absence of a report as an indication of their performance in this or any 
other course in which they are enrolled. 

The following are some resource which students may seek (as appropriate) for assistance in improving 
their performance in their courses: 

(Note: If a student is unsure who is their advisor, they should contact their Dean's Office.) 

Academic Advisor 
Faculty Advisor 
Dean's Office 

Course Instructor/Coordinator 

Learning Center, Address, Phone number 

Counseling Services, Address, Phone number 

Disability Services, Address, Phone number 

Career Counseling, Address, Phone number 

Academic Retention Services, Address, Phone number 

Writing Intensive Tutorial Services, Address, Phone number 

Total Person Program (Student Athletes) Address, Phone number 
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Email Cover Letter #1 



To: Name of Student 

From: The University Council on Advisement 

The University Council on Advisement was created to improve undergraduate academic advising at UU. 
As part of this on-going effort, you have been selected to provide some feedback that will help us to 
enhance academic advising at the University. 

To make the process of completing the survey as easy as possible for you, we have developed a brief 
web-based survey that takes less than five minutes to complete. This survey is located at: [survey website 
address]. It would be greatly appreciated if you could complete the survey and submit it by Thursday, 
November 1 2th. 

Please be assured that your responses will be treated with complete confidentiality and you will never be 
identified individually on any report prepared from this survey. 

Thank you in advance for providing us with information that will help us improve undergraduate advising 
at the University of U. 
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Email Cover Letter Wl 
(sent to non-respondents only) 

To: Name of Student 

From: The University Council on Advisement 

Earlier this week the University Council asked you to complete a brief web-based survey concerning 
advising at UU. If you have recently responded to the survey, then please accept our sincere thanks. If 
not, we encourage you to do so now. 

The University Council is interested in your opinions regarding academic advising to help guide our 
efforts to improve advising for all undergraduate students. This survey will take you less than five 
minutes to respond and is located at: [survey website address]. It would be greatly appreciated if you 
could complete the survey and submit it before Monday, November 16 th . 

Please be assured that your responses will be treated with complete confidentiality and you will never be 
identified individually on any report prepared from this survey. 

Thank you in advance for providing us with information that will help us improve undergraduate advising 
at the University of U. 
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